Image capturing apparatus and image capturing method

ABSTRACT

In a high speed image capturing state, a camera signal processing circuit is not needed to perform a signal process at a high screen rate, but at a regular screen rate. In the high speed image capturing mode, raw data of 240 fps received from an image sensor  101  are recorded on a recording device  111  through a conversion processing section  201  and a recording device controlling circuit  210 . Raw data that have been decimated and size-converted are supplied to a camera signal processing circuit  203  through a pre-processing circuit  202  and an image being captured is displayed on a display section  112  with a signal for which a camera process has been performed. In a reproducing state, raw data are read from the recording device  111  at a low screen rate according to a display performance of the display section  112  and the raw data that have been read are processed are processed by the pre-processing circuit  202  and the camera signal processing circuit  203  and a reproduced image is displayed by the display section  112.

TECHNICAL FIELD

The present invention relates to an image capturing apparatus and animage capturing method, in particular, to those that capture imagesusing a high speed image sensor.

BACKGROUND ART

In the past, high speed video cameras that can capture images at highspeeds have been known. For example, a video camera has accomplishedhigh speed image capturing by converting an image size per one image tobe processed at a high speed rate into ¼ of the standard image size andplacing these four images in an image of the regular rate (see PatentDocument “Japanese Patent Application Laid-Open Publication No. HEI8-88833”). Another video camera has accomplished high speed imagecapturing using a circuit structure that processes data received from asensor in parallel so as to increase a process amount per unit time (seePatent Document “Japanese Patent Application Laid-Open Publication No.HEI 8-251492”).

However, the high speed image capturing described in Patent Document“Japanese Patent Application Laid-Open Publication No. HEI 8-88833” orPatent Document “Japanese Patent Application Laid-Open Publication No.HEI 8-251492” was aimed to temporarily store a captured image in astorage device such as a VTR or a semiconductor memory, reproduce thecaptured image in slow motion, and analyze a very high speed motion andthe device itself had a complicated structure and was expensive. Thus,it was difficult to apply high speed image capturing systems asdescribed in Patent Document 1 or Patent Document 2 to portable imagecapturing devices that have been widespread as home-use devices,so-called camcorders (product names of devices in which a video cameraand a recorder are integrated in one unit), digital cameras, and soforth from view points of portability and power consumption.

With reference to FIG. 1, an image capturing apparatus designed takingaccount of such points will be described. The structure shown in FIG. 1has the same structure as that of an existing camcorder. In other words,an image capturing apparatus 100 shown in FIG. 1 includes an imagesensor 101, a pre-processing circuit 102, a camera signal processingcircuit 103, a conversion processing section 104, a compression anddecompression circuit 105, a memory control circuit 106, a memory 107, adisplay processing circuit 108, a compression and decompression circuit109, a recording device control circuit 110, a recording device 111, adisplay section 112, and a control section 113.

The image sensor 101 can select a high speed image capturing mode inwhich the image sensor 101 reads a signal at a first screen rate (alsoreferred to as frame rate) of 60 fps (fields/second) or more based onthe NTSC specifications or a regular image capturing mode in which theimage sensor 101 reads a signal at a regular second screen rate. Thescreen rate in the high speed image capturing mode is 240 fps that isfour times higher than that of the regular rate. The image sensor 101 isequipped with a CDS (Correlated Double Sampling) and an A/D converterand the image sensor 101 outputs captured image data.

The pre-processing circuit 102 performs an optically correcting processsuch as a shading correction for captured image data that are outputfrom the image sensor 101 and outputs a digital image signal. The camerasignal processing circuit 103 performs a camera signal process such as awhite balance adjustment process for the captured image data that arereceived from the pre-processing circuit 102.

The conversion processing section 104 performs a display decimation anda size adjustment to convert an image signal received from the camerasignal processing circuit 103 into an image signal having a screen rateand a screen size suitable for a display of the display section 112. Thedisplay decimation is performed only when an image signal received fromthe camera signal processing circuit 103 is output to the displayprocessing circuit 108. The display decimation decimates the number offields per unit time of the image signal captured by the image capturingapparatus 100 in the high speed image capturing mode to the number offields per unit time defined in the display standard of the displaydevice (60 fps in this case).

The compression and decompression circuit 105 performs acompression-encoding process for captured image data received from theconversion processing section 104 according to a still image encodingsystem, for example, JPEG (Joint Photographic Experts Group) or thelike. In addition, the compression and decompression circuit 105performs a decompression-decoding process for encoded data of a stillimage supplied from the memory control circuit 106. The memory controlcircuit 106 controls writing and reading image data to and from thememory 107. The memory 107 is a FIFO (First In First Out) type buffermemory that temporarily stores image data received from the memorycontrol circuit 106 and, for example, an SDRAM (Synchronous DynamicRandom Access Memory) or the like is used for the memory 107.

The display processing circuit 108 generates an image signal to bedisplayed on the display section 112 from an image signal received fromthe conversion processing section 104 or the compression anddecompression circuit 109, supplies the signal to the display section112, and causes it to display an image. The display section 112 iscomposed, for example, of an LCD (Liquid Crystal Display) and displays acamera-through image that is being captured or a reproduced image ofdata that have been recorded in the recording device 111.

The compression and decompression circuit 109 performs acompression-encoding process according to a moving image encodingsystem, for example, MPEG (Moving Picture Experts Group) or the like forimage data received from the conversion processing section 104. Inaddition, the compression and decompression circuit 109 performs adecompression-decoding process for encoded data of a moving imagesupplied from the recording device 111 and outputs the resultant data tothe display processing circuit 108. The display section 112 displays amoving image received from the display processing circuit 108.

The control section 113 is a microcomputer composed, for example, of aCPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (RandomAccess Memory), and so forth and totally controls each section of theimage capturing apparatus by executing programs stored in the ROM and soforth.

In the image capturing apparatus shown in FIG. 1, in the high speedimage capturing mode, a captured image signal of 240 fps received fromthe image sensor 101 is supplied to the camera signal processing circuit103 through the pre-processing circuit 102. The conversion processingsection 104 decimates an output image signal of the camera signalprocessing circuit 103 by ¼ so that an output signal of 60 fps isobtained. The decimated and size-converted image signal is supplied tothe display processing circuit 108. The display processing circuit 108generates an image signal to be displayed on the display section 112,supplies the resultant image signal to the display section 112, andcauses it to display an image.

When receiving a record request of a high speed captured image from thecontrol section 113 according to the user's operation, the conversionprocessing section 104 sends an image signal of 240 fps to thecompression and decompression circuit 105. If necessary, the conversionprocessing section 104 reduces the size of the image signal receivedfrom the camera signal processing circuit 103 and sends the image signalto the compression and decompression circuit 105.

The compression and decompression circuit 105 compression-encodes theimage signal received from the conversion processing section 104according to the JPEG format. The memory control circuit 106 temporarilystores encoded data received from the compression and decompressioncircuit 105 into the memory 107. In such a manner, image data for apredetermined period are stored in the memory 107.

When receiving a read request for encoded data stored in the memory 107from the control section 113, the memory control circuit 106 reads theencoded data stored in the memory 107 at 60 fps and sends the encodeddata to the compression and decompression circuit 105. The compressionand decompression circuit 105 decompression-decodes the encoded datareceived from the memory control circuit 106 and sends the decoded datato the conversion processing section 104. When receiving a recordrequest for the recording device 111 from the control section 113, theconversion processing section 104 sends the image signal received fromthe compression and decompression circuit 105 to the compression anddecompression circuit 109. The compression and decompression circuit 109compresses the image signal received from the conversion processingsection 104 according to the MPEG format and stores thecompression-encoded signal to the recording device 111 through therecording device control circuit 110. The conversion processing section104 adjusts the size of the image signal of 60 fps received from thecompression and decompression circuit 105, sends the resultant imagesignal to the display processing circuit 108, and causes display section112 to display a reproduced image.

In the foregoing proposed image capturing apparatus shown in FIG. 1, inthe high speed image capturing mode of the image sensor 101, since thescreen rate of the output captured signal is high, the pre-processingcircuit 102 and the camera signal processing circuit 103 are required tooperate at a high speed. If the system is accomplished by an LSI or thelike, since the calculation scales of the pre-processing circuit 102 andthe camera signal processing circuit 103 are large in the whole system,a high speed process or a parallel process is not advantageous from theview point of circuit area and power consumption. Moreover, in thestructure shown in FIG. 1, since a captured image signal was temporarilystored in the memory 107, there was a problem that after the high speedimage capturing mode was stopped, it took a process time to decode datatemporarily stored in the memory 107 and to re-encode the data accordingto the regular record format.

DISCLOSURE OF THE INVENTION

Therefore, an object of the present invention is to provide an imagecapturing apparatus and an image capturing method that do not need tocause a camera signal processing circuit to perform a process at a highspeed screen rate and that are easy to operate, are produced at lowcost, and thereby have excellent portability.

To solve the foregoing problems, the present invention is an imagecapturing apparatus, comprising:

record medium control means for recording captured image data having afirst screen rate obtained from a solid state image capturing device toa record medium and reading captured image data having a second screenrate which is lower than the first screen rate from the record medium;

conversion means for converting the captured image data having the firstscreen rate obtained from the solid state image capturing device intothe captured image data having the second screen rate;

camera signal processing means for performing a camera signal processfor the captured image data having the second screen rate obtained fromthe conversion means or the record medium control means; and

display processing means for generating an image signal to be displayedfrom an image signal obtained from the camera signal processing means.

The present invention is an image capturing apparatus, comprising:

record medium control means for recording captured image data having afirst screen rate obtained from a solid state image capturing device toa record medium and reading captured image data having a second screenrate which is lower than the first screen rate from the record medium;

conversion means for converting the captured image data having the firstscreen rate obtained from the solid state image capturing device intothe captured image data having the second screen rate;

first camera signal processing means for performing a camera signalprocess for the captured image data having the second screen rateobtained from the conversion means;

second camera signal processing means for performing a camera signalprocess for the captured image data having the second screen rateobtained from the record medium control means; and

display processing means for generating an image signal to be displayedfrom an image signal obtained from either the first and second camerasignal processing means,

wherein the first camera signal processing means is simply structured incomparison with the second camera signal processing means.

The present invention is an image capturing apparatus, comprising:

first record medium control means for recording captured image datahaving a first screen rate obtained from a solid state image capturingdevice to a first record medium and reading captured image data having asecond screen rate which is lower than the first screen rate from therecord medium;

conversion means for converting the captured image data having the firstscreen rate obtained from the solid state image capturing device intothe captured image data having the second screen rate;

first camera signal processing means for performing a camera signalprocess for the captured image data having the second screen rateobtained from the conversion means;

second camera signal processing means for performing a camera signalprocess for the captured image data having the second screen rateobtained from the first record medium control means;

second record medium control means for recording an output image signalof the second camera signal processing means to a second record mediumand reading an image signal from the record medium at the second screenrate; and

display processing means for generating an image signal to be displayedfrom either an output image signal obtained from the first camera signalprocessing means or an output image signal of the second record mediumcontrol means,

wherein the first camera signal processing means is simply structured incomparison with the second camera signal processing means.

The present invention is an image capturing method, comprising:

a record medium control step for recording captured image data having afirst screen rate obtained from a solid state image capturing device toa record medium and reading captured image data having a second screenrate which is lower than the first screen rate from the record medium;

a conversion step for converting the captured image data having thefirst screen rate obtained from the solid state image capturing deviceinto the captured image data having the second screen rate;

a camera signal processing step for performing a camera signal processfor the captured image data having the second screen rate obtained fromthe conversion step or the record medium control step; and

a display processing step for generating an image signal to be displayedfrom an image signal obtained from the camera signal processing step.

The present invention is an image capturing method, comprising:

a record medium control step for recording captured image data having afirst screen rate obtained from a solid state image capturing device toa record medium and reading captured image data having a second screenrate which is lower than the first screen rate from the record medium;

a conversion step for converting the captured image data having thefirst screen rate obtained from the solid state image capturing deviceinto the captured image data having the second screen rate;

a first camera signal processing step for performing a camera signalprocess for the captured image data having the second screen rateobtained from the conversion step;

a second camera signal processing step for performing a camera signalprocess for the captured image data having the second screen rateobtained from the record medium control step; and

a display processing step for generating an image signal to be displayedfrom an image signal obtained from either the first and second camerasignal processing step,

wherein the first camera signal processing step is simply structured incomparison with the second camera signal processing step.

The present invention is an image capturing method, comprising:

a first record medium control step for recording captured image datahaving a first screen rate obtained from a solid state image capturingdevice to a first record medium and reading captured image data having asecond screen rate which is lower than the first screen rate from therecord medium;

a conversion step for converting the captured image data having thefirst screen rate obtained from the solid state image capturing deviceinto the captured image data having the second screen rate;

a first camera signal processing step for performing a camera signalprocess for the captured image data having the second screen rateobtained from the conversion step;

a second camera signal processing step for performing a camera signalprocess for the captured image data having the second screen rateobtained from the first record medium control step;

a second record medium control step for recording an output image signalof the second camera signal processing step to a second record mediumand reading an image signal from the record medium at the second screenrate; and

display processing step for generating an image signal to be displayedfrom either an output image signal obtained from the first camera signalprocessing step or an output image signal of the second record mediumcontrol step,

wherein the first camera signal processing step is simply structured incomparison with the second camera signal processing step.

In the image capturing apparatus according to the present invention,although a screen rate of an image capturing device is high, a camerasignal processing circuit is needed to always satisfy only a screen rateaccording to the display performance of a display section. Thus, it isnot necessary to perform a high frequency drive and a parallel processonly for high speed image capturing and thereby power consumption andcircuit scale can be reduced.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an example of an image capturingapparatus that has been proposed;

FIG. 2 is a block diagram showing an image capturing apparatus accordingto a first embodiment of the present invention;

FIG. 3A and FIG. 3B are schematic diagrams used to describe an exampleof an image sensor according to the present invention;

FIG. 4 is a block diagram showing a more detailed structure of a part ofthe first embodiment of the present invention;

FIG. 5 is a data flow chart in a high speed image capturing modeaccording to the first embodiment of the present invention;

FIG. 6 is a data flow chart in an image reproducing state of the firstembodiment of the present invention;

FIG. 7 is a block diagram showing an image capturing apparatus accordingto a second embodiment of the present invention;

FIG. 8 is a data flow chart in a high speed image capturing mode of thesecond embodiment of the present invention;

FIG. 9 is a data flow chart in an image reproducing state of the secondembodiment of the present invention;

FIG. 10 is a block diagram showing an image capturing apparatusaccording to a third embodiment of the present invention;

FIG. 11 is a data flow chart in a high speed image capturing modeaccording to a third embodiment of the present invention;

FIG. 12 is a data flow chart in an image reproducing state of the thirdembodiment of the present invention;

FIG. 13 is a block diagram showing an image capturing apparatusaccording to a fourth embodiment of the present invention;

FIG. 14 is a data flow chart in a high speed image capturing mode of thefourth embodiment of the present invention; and

FIG. 15 is a data flow chart in an image reproducing state of the fourthembodiment of the present invention.

BEST MODES FOR CARRYING OUT THE INVENTION

Next, with reference to accompanying drawings, a first embodiment of thepresent invention will be described. As shown in FIG. 2, a capturedimage signal received from an image sensor 101 is supplied to an imagecapturing apparatus 200 according to the first embodiment of the presentinvention.

The image sensor 101 converts incident light of an object capturedthrough an optical system (including a lens, an infrared suppressionfilter, an optical low-pass filter, and so forth) into an electricsignal according to the photoelectric conversion. As the image sensor101, for example, a CMOS (Complementary Metal Oxide Semiconductor) typeimage capturing device is used. In the CMOS type image capturing device,photo diodes, line-column selection MOS transistors, signal wires, andso forth are two-dimensionally arranged to form a vertical scanningcircuit, a horizontal scanning circuit, a noise reduction circuit, atiming generation circuit, and so forth. As the image sensor 101, a CCD(Charge Coupled Device) that can capture images at high speeds may beused.

The image sensor 101 can be switched over between a high speed imagecapturing mode in which a signal is read at a first screen rate (alsoreferred to as the frame rate) higher than the regular screen rate (60fps (fields/sec) that is based on the specifications of the NTSC systemand a regular image capturing mode in which a signal is read at a secondscreen rate that is the regular screen rate. The screen rate of the highspeed image capturing mode is needed to be 240 fps that is four timeshigher than that of the regular rate. The image sensor 101 is internallyequipped with a CDS (Correlated Double Sampling), an A/D converter, andso forth and outputs a digitally captured image signal corresponding tothe matrix of pixels of the image sensor 101.

The image sensor 101 uses three image capturing devices that outputcaptured image signals, for example, of three-primary colors and obtainsone output line every four output lines of each image capturing deviceto accomplish a screen rate of 240 fps that is four times higher thanthat of the regular screen rate (60 fps). Assuming that the number ofpixels of one frame at the regular screen rate is, for example, 6.4million pixels, the number of pixels in the high speed image capturingmode is 1.6 million pixels.

FIG. 3A shows an example of an array of color filters according to thepresent invention. A square lattice array is inclined by 45° and each ofR and B filters is surrounded by a G filter. In this structure, whilenecessary and sufficient spatial frequency characteristics are obtainedfor R and B components on the human's visual sensitivity, a spatialfrequency characteristic of G component that is higher than that of eachof R and B components in the human's sensitivities can be improved incomparison with that of the conventional Bayer's array. The G componentbecomes a main component to generate a luminance signal. Thus, not onlythe resolution of luminance of an achromatic object, but that of achromatic object is improved and thereby the image quality is improved.

The color filter array shown in FIG. 3A is based on a method ofalternately reading pixels of two adjacent lines in one horizontalperiod at the regular screen rate as denoted by broken lines. In otherwords, in the regular image capturing mode, pixels are scanned and readin such a sequence.

On the other hand, at the high screen rate, horizontal scanning linesare decimated and read at a rate of one every four horizontal scanninglines. To deal with the high screen rate, as shown in FIG. 3B, assumingthat read outputs of six adjacent column pixels are denoted by CH1 toCH6, respectively, two A/D converters are disposed in common with threealternative columns (CH1, CH3, and CH5 and CH2, CH4, and CH6). The A/Dconverters convert, for example, one sample into digital data of 14bits. Instead, respective A/D converters may be disposed at columnpixels. CDSs (not shown) are disposed in the same manner as the A/Dconverters such that high speed reading can be performed. As the imagesensor 101, another structure in which three-primary color filters arearranged for one image capturing device may be used. Instead, imagecapturing devices using complementary color filters may be used.

The image capturing apparatus 200 includes a conversion processingsection 201, a pre-processing circuit 202, a camera signal processingcircuit 203, a display processing circuit 208, a recording devicecontrol circuit 210, a recording device 111, a display section, and acontrol section 213.

The conversion processing section 201 performs signal shunting anddisplay decimating for a digital image signal received from the imagesensor 101. The display decimating is performed only when a signal isoutput to the display processing circuit 208. The display decimating isa decimation of fields that satisfy the number of fields per unit timedefined in the display standard in the high speed image capturing modeof the image capturing apparatus 200 (in this case, 60 fps).

The pre-processing circuit 202 performs an optically correcting processsuch as a shading correction for a digital image signal that is outputfrom the image sensor 101 and outputs a resultant digital image signal.The camera signal processing circuit 203 performs a camera signalprocess such as a white balance adjustment process (also referred to asa development process, an image creation process, or the like) for theimage signal received from the pre-processing circuit 202. An outputsignal of the camera signal processing circuit 203 is supplied to thedisplay processing circuit 208.

The display processing circuit 208 generates an image signal to bedisplayed on the display section 112 from the image signal received fromthe camera signal processing circuit 203 and supplies the resultantsignal to the display section 112 to cause it to display an image. Thedisplay section 112 is composed, for example, of an LCD (Liquid CrystalDisplay) and displays a camera-through image that is being captured, areproduced image of data recorded on the recording device 111, and soforth. The display section 112 may be disposed outside the imagecapturing apparatus 200 and it may be provided with an interface for anexternal output instead of the display section 112.

The recording device control circuit 210 connected to the conversionprocessing section 201 controls writing and reading image data to andfrom the recording device 111. Data stored in the recording device 111are captured image data that have not been processed by the foregoingpre-processing circuit 202 and camera signal processing circuit 203 andare referred to as raw data in this specification.

As the recording device 111, a magnetic tape, a semiconductor memorysuch as a flash memory, a hard disk, or the like can be used. As therecording device 111, a non-attachable/detachable type is basicallyused. However, the recording device 111 may be attachable/detachablesuch that raw data can be retrieved to the outside. When raw data areretrieved to the outside, raw data that have been processed in thepre-processing circuit 202 are preferably retrieved. The camera signalprocess is performed, for example, according to software of an externalpersonal computer.

The control section 213 is a microcomputer composed, for example, of aCPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (RandomAccess Memory), and so forth and totally controls each section of theimage capturing apparatus by executing programs stored in the ROM and soforth.

FIG. 4 shows an example of a structure of the conversion processingsection 201, the pre-processing circuit 202, and the camera signalprocessing circuit 203. Captured image data received from the imagesensor 101 (including CDSs and A/D converters) are supplied to theconversion processing section 201. The conversion processing section 201is composed of switches SW1 and SW2, a decimation section 221, and asize adjustment section 222. The decimation section 221 performs adisplay decimation. The size adjustment section 222 changes the size ofan image to be displayed to an appropriate size. Either the decimationsection 221 or both of the decimation section 221 and the sizeadjustment section 222 decrease the screen rate of raw data in the highspeed image capturing state to the regular rate. The switches SW1 andSW2 are changed over in the recording state and the reproducing stateand a terminal selected in the recording state is denoted by r and aterminal selected in the reproducing state is denoted by p.

An output image signal of the size adjustment section 222 of theconversion processing section 201 is supplied to a shading correctioncircuit 231 of the pre-processing circuit 202. The shading correctioncircuit 231 corrects the brightness of the vicinity of the screen suchthat it does not become dark. An output signal of the shading correctioncircuit 231 is supplied to the camera signal processing circuit 203.

The camera signal processing circuit 203 is composed, for example, of asimultaneously forming circuit 241, a white balance correction section242, an aperture correction section 243, a gamma correction section 244,and a YC generation section 245 arranged in the order from the inputside. However, the structure of the camera signal processing circuit 203is not limited to that shown in FIG. 4. For example, the arrangementorder of these structural elements may be changed or a part of which maybe omitted.

The simultaneously forming circuit 241 interpolates lost pixels of eachcolor component. The simultaneously forming circuit 241 outputsthree-primary color signals (R, G, B) in parallel. Output signals of thesimultaneously forming circuit 241 are supplied to the white balancecorrection section 242. The white balance correction section 242corrects unbalancing of colors caused by a different color temperatureenvironment of an object and different sensitivities of color filters ofthe sensor.

An output of the white balance correction section 242 is supplied to theaperture correction section 243. The aperture correction section 243 isto perform a contour correction that extracts a portion where a signallargely changes and emphasizes the portion. An output signal of theaperture correction section 243 is supplied to the gamma correctionsection 244.

The gamma correction section 244 corrects input and outputcharacteristics such that the gradation is correctly reproduced when acaptured image signal is output to the display section 112. An outputsignal of the gamma correction section 244 is supplied to the YCgeneration section 245.

The YC generation section 245 generates a luminance signal (Y) and acolor difference signal (C). The luminance signal is generated bycombining the gamma-corrected RGB signals at a predetermined compositionratio. The color difference signal is generated by combining thegamma-corrected RGB signals at a predetermined composition ratio. Thegenerated luminance signal and color difference signal are supplied tothe display section 112 through the display processing circuit 208.

In the image capturing apparatus shown in FIG. 2, in the high speedimage capturing mode, as shown in FIG. 4, the control section 213controls the switches SW1 and SW2 of the conversion processing section201 to select the terminal r side. FIG. 5 shows signal flows in the highspeed image capturing mode. As shown in FIG. 5, raw data of 240 fpsreceived from the image sensor 101 are supplied to the recording device111 through the terminal r of the switch SW1 and the recording devicecontrol circuit 210. When a record request for an image that has beencaptured at a high speed is received from the control section 213according to the user's operation, raw data are recorded on therecording device 111.

The decimation section 221 of the conversion processing section 201decimates raw data by ¼ such that the raw data of 60 fps are obtained.The raw data that have been decimated and size-converted are supplied tothe camera signal processing circuit 203 through the pre-processingcircuit 202. A signal for which the camera signal process has beenperformed in the camera signal processing circuit 203 is supplied to thedisplay section 112 through the display processing circuit 208 and animage that is being captured is displayed on the display section 112.

In the image capturing apparatus shown in FIG. 2, in the reproducingstate, the control section 213 controls the switches SW1 and SW2 of theconversion processing section 201 to select the terminal p side. FIG. 6shows signal flows in the reproducing state. As shown in FIG. 6, rawdata of the low screen rate (for example, 60 fps) are read from therecording device 111 according to the display performance of the displaysection 112 under the control of the recording device control circuit210. The raw data that have been read are supplied from the recordingdevice control circuit 210 to the pre-processing circuit 202 through theswitches SW1 and SW2 of the conversion processing section 201 and thesize adjustment section 222.

An output signal of the pre-processing circuit 202 is supplied to thedisplay section 112 through the camera signal processing circuit 203 andthe display processing circuit 208 and a reproduced image is displayedby the display section 112. For example, when only the screen rate hasbeen changed, the reproduced image becomes a slow motion reproducedimage, the time axis of which has been expanded four times than in therecording state. Instead, images may be captured by changing imagecapturing conditions (exposure condition and so forth) and when they arereproduced, the four types of captured images (any of still images ormoving images) may be compared. Instead, a signal that is read from therecording device 111 may be decimated so as to obtain a frame-by-framereproduction image.

Next, with reference to FIG. 7, an image capturing apparatus 300according to a second embodiment of the present invention will bedescribed. In the image capturing apparatus 300, a compression anddecompression circuit 301 is disposed between a conversion processingsection 201 and a recording device control circuit 210. The compressionand decompression circuit 301 performs a compression-encoding processaccording to an encoding system, for example, JPEG (Joint PhotographicExperts Group) or the like for raw data of the high screen rate receivedfrom the conversion processing section 201. Compression-encoded data arewritten to a recording device 111 under the control of the recordingdevice control circuit 210. As the compression-decompression encodingsystem, a binary data encoding system may be used instead of the JPEG.

FIG. 8 shows signal flows in the recording state. Raw data of the highscreen rate obtained in the high speed image capturing mode arecompression-encoded by the compression and decompression circuit 301 andthe compression-encoded data are recorded on a recording device 111through the recording device control circuit 210. Like the firstembodiment, an image signal whose screen rate has been converted intothe regular screen rate by the conversion processing section 201 andthat has been processed by the pre-processing circuit 202 and the camerasignal processing circuit 203 is displayed on the display section 112.

As shown in FIG. 9, in the reproducing state, encoded data of raw datathat have been read from the recording device 111 and supplied from therecording device control circuit 210 is expansion-decoded by thecompression and decompression circuit 301. Expansion-decoded raw dataare supplied from the conversion processing section 201 to the displaysection 112 through the pre-processing circuit 202, the camera signalprocessing circuit 203, and the display processing circuit 208 and areproduced image is displayed on the display section 112. The read speedof the recording device 111 is set up such that raw data of the regularscreen rate are read.

Next, with reference to FIG. 10, an image capturing apparatus 400according to a third embodiment of the present invention will bedescribed. In the image capturing apparatus 400, a simple pre-processingcircuit 401 and a simple camera signal processing circuit 402 aredisposed downstream of a signal of a conversion processing section 201so as to display a camera-through image that is being captured on thedisplay section 112. “Simple” means that these circuits generate animage that is displayed only on the display section 112 and the imagequality of an image to be displayed may be as low as that satisfying thepurpose of checking for an image of an object being captured. Forexample, the number of bits of a signal displayed on the display section112 is needed to be smaller than that of output data of an A/D converterof the pre-processing circuit 401. The simple pre-processing circuit 401may be omitted. With the simple structure, power consumption and heatgeneration for which an image is monitored in the high speed imagecapturing state can be reduced.

FIG. 11 shows signal flows in the recording state of the image capturingapparatus 400. In the high speed image capturing mode, raw data having ascreen rate of 240 fps received from the image sensor 101 are convertedinto raw data having the regular screen rate of 60 fps by the conversionprocessing section 201. Raw data having the regular screen rate aresupplied to the display processing circuit 208 through the simplepre-processing circuit 401, the simple camera signal processing circuit402, and a terminal r of a switch SW3 and the raw data are displayed bythe display section 112. In the high speed image capturing mode, when arecord command is issued, raw data having the high screen rate arerecorded on the recording device 111 through the recording devicecontrol circuit 210.

As shown in FIG. 12, in the reproducing state, raw data read from therecording device 111 are supplied from the recording device controlcircuit 210 to the pre-processing circuit 202, the raw data areprocessed by the camera signal processing circuit 203, and are sent tothe display section 112 through a terminal p of the switch SW3 and thedisplay processing circuit 208. The pre-processing circuit 202 and thecamera signal processing circuit 203 have the same structure as those ofthe foregoing first and second embodiments and a reproduced image havinga higher quality than an image being captured is displayed on thedisplay section 112.

Next, with reference to FIG. 13, an image capturing apparatus 500according to a forth embodiment of the present invention will bedescribed. The image capturing apparatus 500 has a simple pre-processingcircuit 401 and a simple camera signal processing circuit 402 disposeddownstream of a signal of a conversion processing section 201 so as todisplay a camera-through image being captured on the display section112. Thus, like the foregoing third embodiment, power consumption andheat generation that occur while an image is being monitored can bereduced. In addition, in a non-recording period, raw data can be readfrom a memory 502 at the regular screen rate and a camera signal processcan be performed for the raw data such that the resultant data arerecorded on a recording device 505.

In the image capturing apparatus 500, raw data received from theconversion processing section 201 are supplied to the memory 502 througha memory control circuit 501. The memory control circuit 501 controlswriting and reading image data to and from the memory 502. The memory502 is a FIFO (First In First Out) type memory that temporarily storesimage data received from the memory control circuit 501, for example anSDRAM (Synchronous Dynamic Random Access Memory) or the like. The memory502 performs buffering corresponding to throughputs of thepre-processing circuit 202 and the camera signal processing circuit 203.

An output signal of the camera signal processing circuit 203 is suppliedto a terminal r of a switch SW4. A terminal p of the switch SW4 and aterminal p of a switch SW3 are connected in common. The switch SW4 isconnected to a recording device control circuit 504. Connected to therecording device control circuit 504 is a recording device 505.

The recording device control circuit 504 controls writing and readingimage data to and from the recording device 505 through the switch SW4.Data stored in the recording device 505 are a luminance signal and acolor difference signal processed by the pre-processing circuit 202 andthe camera signal processing circuit 203. The recording device 505 maybe a magnetic tape, a semiconductor memory such as a flash memory, arecordable optical disc, a hard disk, or the like. The recording device505 is basically an attachable/detachable type. Instead, the recordingdevice 505 may not be attachable/detachable type and recorded data maybe output to the outside through a communication interface.

FIG. 14 shows signal flows in the recording state of the image capturingapparatus 500. In the high speed image capturing mode, raw data having ascreen rate of 240 fps received from the image sensor 101 is convertedinto raw data having the regular screen rate of 60 fps by the conversionprocessing section 201. The raw data having the regular screen rate issupplied to a display processing circuit 208 through the simplepre-processing circuit 401, the simple camera signal processing circuit402, and the terminal r of the switch SW3 and the raw data are displayedby the display section 112. When a record command is issued in the highspeed image capturing mode, the raw data having the high screen rate isrecorded on the memory 502 through the memory control circuit 501.

In a recording pause period in the high speed image capturing mode, forexample, in a recoding standby state where a hand is released from arecord button, raw data are read from the memory 502 and the raw dataare processed by the pre-processing circuit 202 and the camera signalprocessing circuit 203 and an output signal of the camera signalprocessing circuit 203 is recorded on the recording device 505 throughthe terminal r of the switch SW4 and the recording device controlcircuit 504. The raw data are read from the memory 502 at the regularscreen rate of 60 fps or a lower rate.

As shown in FIG. 15, in the reproducing state, raw data having theregular screen rate read from the recording device 505 are sent to thedisplay section 112 through the recording device control circuit 504,the terminal p of the switch SW4, and the display processing circuit208. A reproduced image reproduced from the recording device 505 isdisplayed by the display section 112. In the image capturing apparatus500, when the raw data are reproduced, the camera signal process is notrequired in comparison with that of the foregoing embodiment and therebywhen the raw data are reproduced, the power consumption can be reduced.

The present invention is not limited to the foregoing embodiments.Instead, various modifications of the embodiments can be performed basedon the spirit of the present invention. For example, data stored in therecording device 111 of the image capturing apparatus 400 (FIG. 10) ofthe third embodiment or data stored in the memory 502 or the recordingdevice 505 of the image capturing apparatus 500 (FIG. 13) of the fourthembodiment may be compressed.

In addition, the present invention can be applied to devices having animage capturing function such as a mobile phone and a PDA (PersonalDigital Assistants) as well as a camcorder and a digital still camera.In addition, the present invention can be applied to a processing deviceand a recording device for a captured image signal of a small camera fora television phone or a game software application connected to apersonal computer or the like.

1. An imaging apparatus, comprising: an image sensor that includes acolor filter array and is configured to output at a first frame rate afirst number of first frames of image data that have not yet beensubjected to white balance processing; and circuitry configured to: (a)in a recording process, compress the first frames of image data, withoutpreviously performing white balance processing thereon, to generatecompressed image data; store the compressed image data on a storagemedium; and generate a first signal representing a first white-balancedimage sequence that is to be displayed on a display as a camera-throughimage by processing the first frames of image data to generate a secondnumber of white-balanced second frames of image data that is lower thanthe first number of first frames of image data, and by processing thewhite-balanced second frames of image data at a second frame rate whichis lower than the first frame rate; and (b) in a reproduction process,read the compressed image data from the storage medium; decompress thecompressed image data to produce decompressed frames of image data; andgenerate a second signal representing a second white-balanced imagesequence to be displayed on the display as a slow motion reproducedimage by processing the decompressed frames of image data.
 2. Theimaging apparatus of claim 1, wherein the circuitry is furtherconfigured to: in the reproduction process, generate the second signalby processing the decompressed frames of image data at a third framerate which is lower than the first frame rate.
 3. The imaging apparatusof claim 2, wherein the third frame rate is the same as the second framerate.
 4. The imaging apparatus of claim 1, wherein the circuitry isfurther configured to: in the recording process, process the first andsecond frames of image data such that a ratio of the first number offrames to the second number of frames is the same as a ratio of thefirst frame rate to the second frame rate.
 5. The imaging apparatus ofclaim 1, wherein the circuitry is further configured to: in therecording process, generate the first signal such that the firstwhite-balanced image sequence corresponds to at least some of the imagedata that is currently being captured by the image sensor.
 6. Theimaging apparatus of claim 5, wherein circuitry is further configuredto: in the recording process, generate the second number of secondframes of image data by decimating the first number of first frames ofimage data.
 7. The imaging apparatus of claim 1, wherein: the imagesensor is further configured to output the first number of first framesof image data throughout a first time period; and the circuitry isfurther configured to, in the reproduction process, generate the secondsignal so that the second white-balanced image sequence is displayed onthe display throughout a second time period, which is longer than thefirst time period.
 8. The imaging apparatus of claim 1, wherein thecircuitry is further configured to: in the recording process, generatethe second number of white-balanced second frames of image data byconverting the first number of first frames of image data to the secondnumber of non-white-balanced second frames of image data andsubsequently performing white balance processing on the second number ofnon-white-balanced second frames of image data.
 9. The imaging apparatusof claim 8, wherein the circuitry is further configured to: in therecording process, convert the first number of first frames of imagedata to the second number of non-white-balanced second frames of imagedata by decimating the first number of first frames of image data. 10.The imaging apparatus of claim 9, wherein the circuitry is furtherconfigured to: in the recording process, convert the first number offirst frames of image data to the second number of non-white-balancedsecond frames of image data by performing a size adjustment processingon at least some frames of image data.
 11. The imaging apparatus ofclaim 1, wherein the image sensor is configured to be selectivelyswitched between a high-speed image capturing mode in which the imagesensor outputs the first frames of image data at the first frame rate,and a lower-speed image capturing mode in which the image sensor outputsthe first frames of image data at the second frame rate.
 12. The imagingapparatus of claim 1, wherein the second frame rate is one fourth of thefirst frame rate.
 13. The imaging apparatus of claim 1, wherein thecircuitry is further configured to: in the recording process, generatethe second number of white-balanced second frames of image data byperforming optical correction processing on at least some frames ofimage data prior to performing white balance processing on those framesof image data.
 14. The imaging apparatus of claim 13, wherein thecircuitry is further configured to: in the recording process, performthe optical correction processing by performing shading correctionprocessing.
 15. The imaging apparatus of claim 1, wherein the circuitryis further configured to: in the recording process, generate the firstsignal by performing a simultaneous forming process on at least someframes of image data.
 16. The imaging apparatus of claim 15, wherein thecircuitry is further configured to: in the recording process, generatethe first signal by performing aperture correction processing, gammacorrection processing, and YC generation processing on at least someframes of image data.
 17. The imaging apparatus of claim 1, wherein thedisplay comprises a liquid crystal display and the circuitry is furtherconfigured to: in the recording process, process the white-balancedsecond frames of image data by performing display processing on thoseframes of image data to generate the first signal, and provide the firstsignal to the liquid crystal display so as to cause the liquid crystaldisplay to display the first white-balanced image sequence.
 18. Theimaging apparatus of claim 1, wherein the circuitry comprises: firstcircuitry configured to, in the recording process, generate the firstsignal; and second circuitry configured to, in the reproduction process,generate the second signal, wherein at least some of the secondcircuitry is not included in the first circuitry.
 19. The imagingapparatus of claim 18, wherein the first circuitry is configured toperform a different simultaneous forming process than the secondcircuitry.
 20. The imaging apparatus of claim 18, wherein the firstcircuitry is configured to generate the first signal such that the firstwhite-balanced image sequence represented thereby is of a lower qualitythan the second white-balanced image sequence represented by the secondsignal generated by the second circuitry.
 21. The imaging apparatus ofclaim 1, wherein the image sensor is further configured to generate thefirst frames of image data at a frame rate of approximately 240 framesper second.
 22. The imaging apparatus of claim 1, wherein: the storagemedium is configured to be attachable to and detachable from the imagingapparatus so as to allow the compressed image data to be stored on thestorage medium when the storage medium is attached to the imagingapparatus and to be processed by a device separate from the imagingapparatus when the storage medium is detached from the imagingapparatus.
 23. The imaging apparatus of claim 1, wherein the circuitryis further configured to: in the recording process, compress the firstframes of image data, without previously performing simultaneous formingprocessing thereon, to generate the compressed image data stored on thestorage medium.
 24. The imaging apparatus of claim 1, wherein thecircuitry is further configured to: in the recording process, compressthe first frames of image data, without previously performing gammacorrection processing thereon, to generate the compressed image datastored on the storage medium.
 25. The imaging apparatus of claim 1,wherein the circuitry is further configured to: in the recordingprocess, compress the first frames of image data, without previouslyperforming YC generation processing thereon, to generate the compressedimage data stored on the storage medium.
 26. The imaging apparatus ofclaim 1, wherein the circuitry is further configured to: in therecording process, compress the first frames of image data, withoutpreviously performing aperture correction processing thereon, togenerate the compressed image data stored on the storage medium.
 27. Theimaging apparatus of claim 1, wherein the circuitry is furtherconfigured to: in the recording process, compress the first frames ofimage data, without previously performing optical correction processingthereon, to generate the compressed image data stored on the storagemedium.
 28. The imaging apparatus of claim 1, wherein the circuitry isfurther configured to: in the recording process, compress the firstframes of image data, without previously performing shading correctionprocessing thereon, to generate the compressed image data stored on thestorage medium.
 29. The imaging apparatus of claim 1, wherein thecircuitry is further configured to: in the recording process, compressthe first frames of image data by encoding each frame according to apredetermined format for each frame to generate the compressed imagedata stored on the storage medium.
 30. The imaging apparatus of claim29, wherein the predetermined format for each frame is JPEG (JointPhotographic Experts Group) format.
 31. The imaging apparatus of claim1, wherein the circuitry is further configured such that, in therecording process, the image data that is compressed and stored on thestorage medium is RAW image data.
 32. The imaging apparatus of claim 1,wherein the color filter array comprises an RGB color filter array. 33.The imaging apparatus of claim 10, wherein the circuitry comprises amicrocomputer.
 34. An imaging apparatus, comprising: an image sensorthat includes a color filter array and is configured to output at afirst frame rate a first number of first frames of image data that havenot yet been subjected to white balance processing, gamma correctionprocessing, and YC conversion processing; and circuitry configured to:(a) in a recording process, compress the first frames of image data,without previously performing white balance processing, gamma correctionprocessing, and YC conversion processing thereon, to generate compressedimage data; store the compressed image data on a storage medium; andgenerate a first signal representing a first white-balanced,gamma-corrected, and YC converted image sequence that is to be displayedon a display as a camera-through image by processing the first frames ofimage data to generate a second number of second frames of image datathat is lower than the first number of first frames of image data, andby performing at least one of white balance processing, gamma correctionprocessing, or YC conversion processing on the second frames of imagedata at a second frame rate which is lower than the first frame rate;and (b) in a reproduction process, read the compressed image data fromthe storage medium; decompress the compressed image data to producedecompressed frames of image data; and generate a second signalrepresenting a second white-balanced, gamma-corrected, and YC convertedimage sequence to be displayed on the display as a slow motionreproduced image by processing the decompressed frames of image data.35. The imaging apparatus of claim 34, wherein the circuitry is furtherconfigured to: in the reproduction process, generate the second signalby processing the decompressed frames of image data at a third framerate which is lower than the first frame rate.
 36. The imaging apparatusof claim 34, wherein the third frame rate is the same as the secondframe rate.
 37. The imaging apparatus of claim 34, wherein the circuitryis further configured to: in the recording process, process the firstand second frames of image data such that a ratio of the first number offrames to the second number of frames is the same as a ratio of thefirst frame rate to the second frame rate.
 38. The imaging apparatus ofclaim 34, wherein the circuitry is further configured to: in therecording process, generate the first signal such that the firstwhite-balanced, gamma-corrected, and YC converted image sequencecorresponds to at least some of the image data that is currently beingcaptured by the image sensor.
 39. The imaging apparatus of claim 38,wherein circuitry is further configured to: in the recording process,generate the second number of second frames of image data by decimatingthe first number of first frames of image data.
 40. The imagingapparatus of claim 34, wherein: the image sensor is further configuredto output the first number of first frames of image data throughout afirst time period; and the circuitry is further configured to, in thereproduction process, generate the second signal so that the secondwhite-balanced, gamma-corrected, and YC converted image sequence isdisplayed on the display throughout a second time period, which islonger than the first time period.
 41. The imaging apparatus of claim34, wherein the circuitry is further configured to: in the recordingprocess, convert the first number of first frames of image data to thesecond number of second frames of image data by decimating the firstnumber of first frames of image data.
 42. The imaging apparatus of claim41, wherein the circuitry is further configured to: in the recordingprocess, convert the first number of first frames of image data to thesecond number of second frames of image data by performing a sizeadjustment processing on at least some frames of image data.
 43. Theimaging apparatus of claim 34, wherein the image sensor is configured tobe selectively switched between a high-speed image capturing mode inwhich the image sensor outputs the first frames of image data at thefirst frame rate, and a lower-speed image capturing mode in which theimage sensor outputs the first frames of image data at the second framerate.
 44. The imaging apparatus of claim 34, wherein the second framerate is one fourth of the first frame rate.
 45. The imaging apparatus ofclaim 34, wherein the circuitry is further configured to: in therecording process, generate the first signal by performing opticalcorrection processing on at least some frames of image data prior toperforming the at least one of white balance processing, gammacorrection processing, or YC conversion processing on those frames ofimage data.
 46. The imaging apparatus of claim 45, wherein the circuitryis further configured to: in the recording process, perform the opticalcorrection processing by performing shading correction processing. 47.The imaging apparatus of claim 34, wherein the circuitry is furtherconfigured to: in the recording process, generate the first signal byperforming a simultaneous forming process on at least some frames ofimage data.
 48. The imaging apparatus of claim 47, wherein the circuitryis further configured to: in the recording process, generate the firstsignal by performing each of white balance processing, gamma correctionprocessing, and YC conversion processing on the second frames of imagedata.
 49. The imaging apparatus of claim 48, wherein the circuitry isfurther configured to: in the recording process, generate the firstsignal by performing aperture correction processing on the second framesof image data.
 50. The imaging apparatus of claim 34, wherein thedisplay comprises a liquid crystal display and the circuitry is furtherconfigured to: in the recording process, generate the first signal byperforming display processing on the second frames of image data, afterperforming the at least one of white balance processing, gammacorrection processing, and YC conversion processing thereon, andproviding the first signal to the liquid crystal display so as to causethe liquid crystal display to display the first white-balanced,gamma-corrected, and YC converted image sequence.
 51. The imagingapparatus of claim 34, wherein the circuitry comprises: first circuitryconfigured to, in the recording process, generate the first signal; andsecond circuitry configured to, in the reproduction process, generatethe second signal, wherein at least some of the second circuitry is notincluded in the first circuitry.
 52. The imaging apparatus of claim 51,wherein the first circuitry is configured to perform a differentsimultaneous forming process than the second circuitry.
 53. The imagingapparatus of claim 51, wherein the first circuitry is configured togenerate the first signal such that the first white-balanced,gamma-corrected, YC converted image sequence represented thereby is of alower quality than the second white-balanced, gamma-corrected, YCconverted image sequence represented by the second signal generated bythe second circuitry.
 54. The imaging apparatus of claim 34, wherein theimage sensor is further configured to generate the first frames of imagedata at a frame rate of approximately 240 frames per second.
 55. Theimaging apparatus of claim 34, wherein: the storage medium is configuredto be attachable to and detachable from the imaging apparatus so as toallow the compressed image data to be stored on the storage medium whenthe storage medium is attached to the imaging apparatus and to beprocessed by a device separate from the imaging apparatus when thestorage medium is detached from the imaging apparatus.
 56. The imagingapparatus of claim 34, wherein the circuitry is further configured to:in the recording process, compress the first frames of image data,without previously performing simultaneous forming processing thereon,to generate the compressed image data stored on the storage medium. 57.The imaging apparatus of claim 34, wherein the circuitry is furtherconfigured to: in the recording process, compress the first frames ofimage data, without previously performing white balance processingthereon, to generate the compressed image data stored on the storagemedium.
 58. The imaging apparatus of claim 34, wherein the circuitry isfurther configured to: in the recording process, compress the firstframes of image data, without previously performing gamma correctionprocessing thereon, to generate the compressed image data stored on thestorage medium.
 59. The imaging apparatus of claim 34, wherein thecircuitry is further configured to: in the recording process, compressthe first frames of image data, without previously performing YCconversion processing thereon, to generate the compressed image datastored on the storage medium.
 60. The imaging apparatus of claim 34,wherein the circuitry is further configured to: in the recordingprocess, compress the first frames of image data, without previouslyperforming aperture correction processing thereon, to generate thecompressed image data stored on the storage medium.
 61. The imagingapparatus of claim 34, wherein the circuitry is further configured to:in the recording process, compress the first frames of image data,without previously performing optical correction processing thereon, togenerate the compressed image data stored on the storage medium.
 62. Theimaging apparatus of claim 34, wherein the circuitry is furtherconfigured to: in the recording process, compress the first frames ofimage data, without previously performing shading correction processingthereon, to generate the compressed image data stored on the storagemedium.
 63. The imaging apparatus of claim 34, wherein the circuitry isfurther configured to: in the recording process, compress the firstframes of image data by encoding each frame according to a predeterminedformat for each frame to generate the compressed image data stored onthe storage medium.
 64. The imaging apparatus of claim 63, wherein thepredetermined format for each frame is JPEG (Joint Photographic ExpertsGroup) format.
 65. The imaging apparatus of claim 34, wherein thecircuitry is further configured such that, in the recording process, theimage data that is compressed and stored on the storage medium is RAWimage data.
 66. The imaging apparatus of claim 34, wherein the colorfilter array comprises an RGB color filter array.
 67. The imagingapparatus of claim 34, wherein the circuitry is further configured to:in the recording process, generate the first signal by performing whitebalance processing on the second frames of image data at the secondframe rate.
 68. The imaging apparatus of claim 34, wherein the circuitryis further configured to: in the recording process, generate the firstsignal by performing gamma correction processing on the second frames ofimage data at the second frame rate.
 69. The imaging apparatus of claim34, wherein the circuitry is further configured to: in the recordingprocess, generate the first signal by performing YC conversionprocessing on the second frames of image data at the second frame rate.70. The imaging apparatus of claim 34, wherein the circuitry comprises amicrocomputer.